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ABSTRACT 

Evaluation can be crucial to help meet the needs and 
requirements of the teacher, students, and curriculum in distance 
education. Distance education usually encompasses a wide diversity of 
students, multi-site classrooms, little opportunity for interaction 
among students, and unfamiliar technology. Because of these 
uncertainties, evaluation of both courses and programs will provide 
teachers with information that can help maJce instruction effective. 
Formative evaluation taJces place during all stages of instruction and 
is designed to promote revision of procedures and/or materials. 
Summative evaluation taJces place at the end of instruction and is 
designed to assess the overall effectiveness of the instruction. A 
quantitative evaluation yields a breadth of information, and asks 
questions that can be statistically tabulated and analyzed. A 
qualitative evaluation gathers a wider range and depth of information 
using open-ended questions, interviews, and observations. A 
successful evaluation entails reviewing course goals and objectives, 
knowing students' needs, choosing the information to be gathered, 
collecting and analyzing data, and incorporating the evaluation 
findings in the revision. This brief guide begins with a statement on 
the importance and purpose of evaluation in distance education. It 
then presents guidelines for six steps in the evaluation process: (1) 
planning evaluation strategy; (2) reviewing course goals and 
objectives; (3) knowing students and their needs; (4) deciding what 
information should be gathered; (5) collecting and analyzing data; 
and (6) using the results to revise instruction. (DB) 
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EVALUATION— VITAL TO DISTANCE EDUCATION 



Distance education is an emerging field open to innovation, challenge, c\nd development. 
As a distance educator, 

Vnu no longer have : 

• A traditional, familiar classroom 

• A relatively homogeneous group of students 

• Face-to-face feedback during class 

• Total control over the medium used in your teaching 

• Ease and convenience for otudent conferencing 
Now, you have: 

• A multi-site "classroom" 

• A wide diversity of students 

• Learners who have little opportunity for interaction with fellow students 

• An unfamiliar technology that may fight you 

• Obstacles for student support. 

The purpose of evaluation... 

From an instructor's perspective, you evaluate to determine: 

• Whether the course objectives are being met. 

• What works, what doesn't. 

• How well the students are doing. 

• If and how the course can be improved. 

Evaluation can he a kev to heln meet th e needs and requirements of the instructor, 
fltuo' ents. and curriculum. 

Why evaluate... 

Course and program evaluation will: 

• Give you control in creating a better program. 

• Provide you with information to help you make instruction, materials, and 

processes as effective as possible. 

• Allow student involvement in helping them accomplish their objectives. 

• Help to determine whether or not growth and change are taking place. 

• Add to the information on effective distance delivered education. 



To think about... 

• How will I know if the students are getting the course content they need? 

. how effective am I as a facilitator, lecturer, counselor, ©valuator, discussion leader? 
■ is everyone comfortable using the equipment? 

• Are the assignments overwhelming or mundane? 

• Is there clarity of content? 

• Is class time well spent? 

• What student information do I want? Age, occupation, reasons for taking the class... 

• Will the evaluative information I gather enable me to make the course more interesting and 

relevant? 



STEPS TO EFFECTIVE EVALUATION 

1. PLAN YOUR EVALUATION STRATEGY 

The most effective evaluation strategy incorporates 
FORMATIVE and SUMMATIVE evaluation. 

«» Formative evaluation: 

• Is an on-going process to be considered at all stages of instruction. 

• Will enable you to improve y our course as you proceed. 

• Facilitates course and content adaptation. 

• Will identify major gaps in your instructional plan or the need for minor 
adjustments. 

• Is an essential pre-condition for summative evaluation. 

Ideas for formative data collection: 

• Post cards— Provide each student with cards, stamper; and addressed already. 

—Ask weekly that they share their own concerns or ;espond to your questions during the last 
three to five minutes of class. 

• Electronic mail (FAX)— Arrange implementation, introduce and instruct in its use. 

• Computer/modem— For the sites in which this technology is available, arrange for its use, too. 

• Telephone— Call often. 

—Start out with an open ended question (Such as, What snags did you run into with the 
second writing assignment? ) to let students discuss their concerns. 
—Follow with probes [such as, Then, will you need more information sources?]. 
—When needing to ask a particular question to gather specific information, ask that last. 

• Round table statements or questions— Use at the beginning or end of class. 

—Check students' need for assistance and possible follow-up, clarification of terms, 
assignments, problems, setting individual appointments for later,... 

• Phone-in office hours —Set them, but be sure to welcome calls at other times. 



«« Summative evaluation: 

• Assesses overall effectiveness of finished product and processes. 

• Can be a springboard in developing a revision plan. 

• Can be a base-line of information for designing a new plan or program. 

• Will not help current students since it is conducted after the course. 



Suggested Feedback Questions 

1 . List five weaknesses of the course. 

2. List three (or five) strengths of the course. 

3. If you were teaching the course, what would you do differently? 

4. Demographic information of your choice: age, level in school, course name, number of 

distance delivered courses taken prior to this one, ... 

5. What would you recommend to a friend planning to take this course? 

6. What did you think would be covered in this course but was not? 

7. Would you recommend this course to a friend? Why or why not? 

8. Brainstorm solutions to problem areas. 
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The best plan should also have DEPTH and BREADTH. •• 

For BREADTH... 
Quantitative evaluation 

• Involves asking questions which can be statistically tabulated and analyzed, 
frequently using a scale, check list, or yes/no responses. 



For example, " Rate the usefulness of assignment #3." 




very useful 


little use 




1 2 3 


4 


5 



• Limits students to responding to the categ ories vou made available to them. 

• Needs a large student sample for relevant statistical analyses. 

Beware: Numbers can give the illusion of precision. 



For DEPTH... 

» Qualitative evaluation: 

• Involves gathering a wirier rang e and depth of information 

• Is more difficult to tabulate into neat categories. 

• Can use open ended questions, participant and non-participant observation 

and interviews, either by phone or in person. 



For example, "What made Assignment #3 useful or not useful to you?" 



• Will not be affected by typical small class size. 

• Is a more flexible and dynamic form of evaluation. 

• Is not limited to your pre-conceived topics of inquiry. 

• Allows for student input of topics. 

Beware: Objectivity is not assured. 

2. REVIEW COURSE GOALS AND OBJECTIVES 

• What are the required program goals and objectives? 

• What are the goals and objectives of the sludsnis? 

• Why is instruction needed? Is this course part of a larger program? 



3. KNOW YOUR STUDENTS AND THEIR NEEDS 

• What is their interest level? 

• How large is your audience? 

• What differences are there in learner experience, age, culture, environment, 

education...? 

• What is their background in the content to be covered? 

• How familiar are they with the instructional methods and technology to be used? 



4. DECIDE WHAT INFORMATION YOU WANT TO GATHER 



• In what areas are problems likely? 

(For example: use of technology, contact and assistance from the instructor, class format and 
interaction, materials, assignment feedback, range of abilities...) 

• What distance delivery restraints are placed on the instructor/students? 

• What are the effects of those restraints? 

• Will you ultimately adapt content, program, or methods to meet student needs, or will 

the learner be expected to adapt to the program requirements? 



Consider the following areas for inquiry: 

• Use of the technology— familiarity, concerns, problems, positive aspects, 

attitude toward technology 

• Class format— effectiveness of lecture, discussion, question and answer 

• Class climate— quantity and quality of interaction with other students and with 

instructor; conduciveness of class atmosphere to student learning; quality 
of questions or problems raised in class; encourac^ment given students 
to express themselves 

• Course content— relevancy, adequate body of knowledge, organization 

• Assignments— usefulness, degree of difficulty and time required, timeliness of 

feedback, readability level of print materials 
. Tests— frequency, relevancy, sufficient review, difficulty, feedback 

• Support services— facilitator, technology, library services, instructor availability 

• Student circumstances— time required and available, constraints, goals, 

motivation 

• Student achievement— adequacy, appropriateness, timeliness, student 

involvement 

• Student Attitude— attendance, assignments submitted, class participation 

• Instructor— contribution as discussion leader, effectiveness, organization, 

preparation, enthusiasm, openness to student views 



5. COLLECT AND ANAL YZE YOUR DA TA 

Because your student numbers will no doubt be small, getting a 100% 
survey return is crucial!! 

• Encourage cooperation in the evaluation process. 

• Stress your interest and need to receive their feedback in order to inake 

changes. 

• Make your evaluations of the course quick and easy for students to do— you 

will get better response that way. 

• Vary the evaluation method as needed or try different ways to see which suits 

you and individual students better 

• Provide in-class time for completing evaluation forms. 
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6 USE WHA T YOU KNOW TO REVISE INSTRUCTION .: 

lncQrporate_y our evaluative tindjn^s; * _ • . . .; • 

. . . . Revision is an important way to make q £ood;cpuFse better:* : :% 

• - What did your students tell you? Have you considered^ ondismissed it?.. • ' 
. If student performance is not up to expectation, mak£ changes. 
Are*there ways the content can.be-made mofe, adaptive to individual students? 
. ' • Consider; , retnforceinent. review, repetition, remediation. - . 
•^•i^youLCouise^ '.. . • 

• Develop \ to the point where.it junctions as effectively as , po.ssibla--t{ieajps£ 

••• sum^itive^ ■* " * * ' ' *• 

* Share the relevantigsuliS of your evaluation with other instructors. \ 



Evaluation Tips » 

/ .• • . ; - ■ • •« . . . 

• Check out already published questionnaires: there's no need to re- invent the 

.• wheel. •■' , ■ ' . •• • ' 

''• Draft and revise your questions: change if necessary. ■ ,■ ■. ■•. . ■• ^ • .' 

-•♦ Make use of foliow.Qp.prObes—ask more questions in areas of indicated concern. 

•* Sequence your questions for best effect— go ahead and ask. for. suggestions, for 

* improvement before asking for what is good. {It wtfhelp convey the-message . 
' that you are sincere about seeking improvements,) - • . ~ ) - 

' • PJace.opeh ended questions after quick answer questions. (Students will £ave .. 

•*■ buitt-Ui thinking- time to assist-in giving longer, thoughtful answers.) ■■ : , 
: • Onsummative 'evaluation, assure anonymity, • '.• : 

* . " • • (You could accomplish this by having all questionnaires sent to a neutral site 
. - where they would be removed from their, envelopes, and forwarded to you 
without postmark.) .. v ~ ■< ' : «* * * '■■<■ ' = 

> Make use' of informal evaluation—its not necessary to use a formal evaluation • 

tool alUhfe time. ■ ■ , ■ ', ' ' . •' ( ' : * ■• V 

•■• Establish rapport- b& bemg ^e.tesfe.d. and supportive. Withhold ju<§TOqtal .. ;/••.. 

• responses. ' '. •• . • ■ .' w V * > 

' • Adapt to the student in degree of formality and pace $ ■communication. . •■; •. v 



r \ 4 > IrTsummary... m - ^ .\; / ^ 

..' ; ' . • P (ardour evaluation strategy, 

• fiettfewcourse goals and objectives.' ♦ 

Know your students and their needs. •' 
•.Decide what information you want to gather, 
. # •' • Collect and analyze your s data,. ( 
v . •■Use what youkriowtg revise instruction •' ; s 



FOR MORE INFORMATION: 

..'..*• • * • • • 

Statewide Distance Education 
; University ot Alaska System ,..-.= 
! 3890 University Lake Drive ' 
Anchorage. -Alaska. 99508 ■ ' . ' 





